Controller and control method for robotic system

ABSTRACT

A controller for a robotic system is provided. The controller may derive first data including information about operation objects and information about pallet before loading and/or unloading of the operation object on/from the pallet. The controller may derive a control sequence for loading and/or unloading the operation object on/from the pallet based on the first data before execution of the corresponding loading/unloading task. Based on the control sequence, the controller may instruct a robot to execute the task of loading and/or unloading the operation object on/from the pallet.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.16/886,734, filed May 28, 2020, issued as U.S. patent Ser. No. ______,which claims priority and the benefit of Japanese Patent Application No.2019208587 filed Nov. 19, 2019. The disclosure of each of theseapplications is incorporated by reference herein in its entirety. U.S.patent application Ser. No. 16/886,734 is a continuation-in-part of thefollowing applications: U.S. patent application Ser. No. 16/428,870,filed May 31, 2019, issued as U.S. Pat. No. 10,647,528 on May 12, 2020,U.S. patent application Ser. No. 16/428,645, filed May 31, 2019, nowissued as U.S. Pat. No. 10,696,493 on Jun. 30, 2020, U.S. patentapplication Ser. No. 16/428,843, filed May 31, 2019, now issued as U.S.Pat. No. 10,696,494 on Jun. 30, 2020, U.S. patent application Ser. No.16/428,714, filed May 31, 2019, issued as U.S. Pat. No. 10,679,379 onJun. 9, 2020, and U.S. patent application Ser. No. 16/428,809, filed May31, 2019, issued as U.S. Pat. No. 10,618,172 on Apr. 14, 2020, all ofwhich are incorporated by reference herein in their entireties.

TECHNICAL FIELD

The present disclosure generally relates to a robotic system, and moreparticularly to a controller (control apparatus) for a robotic systemthat manipulates objects (objects such as items) as well as to a roboticsystem control method, a logistics system (physical distributionsystem), a program, and a recording medium.

BACKGROUND

Many robots (e.g., machines configured to automatically/independentlyperform physical operation) currently enjoy wide use in many fieldsbecause of their ever-improving performance and falling costs. Forexample, robots can be used to perform various operations such asmaneuvering and movements of objects in production, assemblage, packing,transfer, transportation, and the like. In performing operations, robotscan repeat human motions, and thereby take over or reduce dangerous orrepetitive human operations.

As a system (robotic system) using such robots, for example, JapanesePatent Laid-Open No. 2018-167950 proposes an automatic logistics systemequipped with a transport container storage mechanism adapted totemporarily store a transport container for automation and laborsavingof processes from warehousing to shipping of items and an automatic itemdelivery mechanism adapted to automatically concentrate items from thetransport container in a shipment container based on shippinginformation.

However, in spite of technological advancement, robots often lacksophistication needed to reproduce a human-involved operation incarrying out a larger and/or more complicated task. Therefore, manyautomation and functional expansion of robotic systems are stillinsufficient and still require human involvement. It is also difficultto sufficiently enhance handling property and storage efficiency forobjects of different attributes for example. Consequently, there isstill a need for technological improvements in order to manage variousmotions and/or interactions among robots and further promote automationand functional expansion of the robotic systems. Thus, one object of thepresent technology is to provide a robotic system controller and arobotic system control method that implement sophisticated cooperationamong robotic units and enhance handling and storage efficiency forvarious types of objects.

SUMMARY

To solve the above problem, the present invention adopts the followingconfigurations.

[1] A controller according to the present technology controls a roboticsystem including at least one operation robot adapted to manipulate anoperation object to load the operation object on a pallet (palletize)and/or unload the operation object from the pallet (depalletize), wherethe operation robot manipulate the operation object using a robot arm,an end effector, and the like. Some examples of the operation robot caninclude a palletizing robot, a depalletizing robot, a piece-pickingrobot, a fetching robot, a devanning robot, and the like. The controllercan include (1) a data deriving unit adapted to derive first dataincluding information about an operation object and information about apallet before loading of the operation object on the pallet and/orunloading of the operation object from the pallet; (2) a data storageunit adapted to store the first data; and (3) a robot control unitadapted to create or derive a control sequence for loading the operationobject on the pallet and/or unloading the operation object from thepallet based on the first data. Based on the control sequence, thecontroller can instruct the operation robot to execute a task forloading of the operation object on the pallet and/or for unloading ofthe operation object from the pallet.

Here, the “operation object” may include an object manipulated by theoperation robot provided in the robotic system. The operation object mayinclude, for example, one or more items as well as a container such as abin, a storage container, or a box, which contains the items or has theitems placed thereon. The container may be either packed or unpacked.Also, part of (e.g., a top face) the container may be open. Also, insome embodiments, the “operation object” may be an object stored at ashelf, a pallet, a conveyor, and a temporary storage area. The “pallet”can include a loading table having a surface on which objects are placedin a known quantity for loading, transport, and storage. Examples of thepallet include a plane pallet, a box pallet, a post pallet, and thelike. The pallet may or may not have an insertion port such as a folk.Furthermore, the “information about the object” and “information aboutthe pallet” may include identification information for the operationobject and identification information for the pallet, respectively.Also, the “control sequence” may include an operating sequence derivedin advance for controlling or causing one or more units (e.g., robots)in the robotic system to execute individual tasks. Besides, the term“before” means that something is taking place at or prior to a giventime point and the term “after” means that something is taking place ator subsequent to a given time point.

The “information about the operation object” includes at leastpositional information (e.g., pose, location, and/or orientation) of theoperation object and information about “stock keeping unit (SKU)” of theoperation object. The “stock keeping unit (SKU)” is a minimum managementunit for performing order and stock management of the operation objects.For example, operation objects belonging to one type or category may bedistinguished depending on one or more differences in attribute, such aspackaging, content size, content color, and/or content quantity. Forexample, one type or category of operation objects may include aparticular brand/design of clothes but have three colors and four sizes(e.g., S, M, L, and XL). Accordingly, one type/category of the operationobjects may be further categorized into 12 SKUs. Note that SKU can bedynamically set by an operator. SKU may be set without distinguishingthe objects by the color or size. In general, however, it is consideredthat operation objects may be distinguished by the SKU according toexample factors described below (the Distribution Systems ResearchInstitute, GTIN Guidelines, Revised Edition of GTIN AllocationGuidelines Version 2018.12, p.10).

-   -   There is any element that must be determined to be a different        item in sales;    -   It is desirable to appeal the differences in the goods to the        consumers;    -   It is desirable to analyze the sales of the goods individually.        (Examples of different attributes within a type/category of        goods)    -   Different size (e.g., large, medium, small);    -   Difference in the amount of packaged content (e.g., weight,        quantity, the number of pieces, etc.);    -   Different packaging form (e.g., bag, can, bottle, etc.);    -   Different appearance (e.g., color);    -   Different flavor;    -   Different scent;    -   Different sales units (e.g., groupings of 3 pieces, 5 pieces, 15        pieces, etc.); and/or    -   Different packaging combination of goods (e.g., gift baskets).

With such example configuration, information about the operation objectand information about the pallet may be obtained before loading theoperation object on the pallet and/or before unloading the operationobject from the pallet. Accordingly, a task may be executed based oncreating the control sequence according to the first data including suchinformation. Since the control sequence may be derived before execution,the loading/unloading tasks may be executed efficiently and smoothlywithout breaks resulting from reactive computations of the sequence(e.g., for the next action/portion) during execution of the overalltask.

[2] In the above configuration, the data deriving unit may derive seconddata including the information about the operation object and theinformation about the pallet after the operation object is loaded on thepallet and/or the operation object is unloaded from the pallet. The datastorage unit may store the second data.

With this configuration, information about the operation object andinformation about the pallet can be obtained after the operation objectis loaded on the pallet and/or after the operation object is unloadedfrom the pallet. This makes it possible to track a state or situation ofthe pallet reliably after the operation object is loaded and/or afterthe operation object is unloaded.

[3] In the above configuration, the robot control unit may be adopted tocreate or derive the control sequence based on the first data and/or thesecond data such that a plurality of types of operation objects (e.g.,different SKUs) are loaded on the pallet.

With this configuration, SKUs of (1) the operation objects scheduled tobe loaded on the pallet and/or (2) the operation objects having beenloaded on the pallet can be identified from the first data and thesecond data. Therefore, even in the case where different operationobjects and/or different SKUs are loaded in the same pallet (such as formixed loading), it is possible to monitor and trace the loaded state ofthe operation objects. This makes it possible to increase the loadfactor of the pallet, thereby improving the utilization efficiency ofthe pallet. Moreover, since filling information/availability informationof the pallet can be determined, as the next stage, selection of apallet for loading the operation object on the pallet and/or unloadingthe operation object from the pallet can be performed efficiently. Thisalso makes it possible to prevent the pallet from being fragmented withunused spaces (defragmentation), and makes it easier to collectivelymanage positional information about the operation object and the pallet.

[4] In the above configuration, the robot control unit may be adapted tocreate or derive the control sequence based on the first data and thesecond data such that the operation objects of the same or similar stockkeeping units (SKUs) are placed in a concentrated manner on the pallet.Here, “similar” means that among the attributes (color, size, and thelike as described above) defining the SKUs, a predetermined number ormore of the attributes are common. The present configuration also allowsplacement and tracking of operation objects of the same or similar SKUson the same pallet (for non-mixed loading) and/or operation objects ofdifferent SKUs are loaded on the same pallet (for mixed loading). In thecase of mixed loading, operation objects of the same or similar SKUs onthe pallet can be loaded in a concentrated manner (e.g., stacked in thevertical direction, placed adjacent to each other or the like).

With this configuration, the SKUs of the operation objects targeted tobe loaded on the pallet and the operation objects previously loaded onthe pallet can be identified and processed (e.g., simulated forplacement/motion planning) using the first data and the second data.This makes it easy to place the operation objects of the same or similarSKUs in a concentrated manner. Consequently, regardless of single-typeloading or mixed loading, the load factor can be increased, and theutilization efficiency of the pallet can be improved. Moreover, sincethe operation objects having the same or similar SKUs are placed in aconcentrated manner, as the next stage, selection of a pallet forloading the operation object on the pallet and/or unloading theoperation object from the pallet can be performed efficiently. In otherwords, creating or deriving the control sequence may include determiningor selecting, based on the first data and the second data, one of thepallets on which the operation object is to be loaded and/or uploaded.

[5] In the above configuration, the robot control unit may compute,based on the second data, an index indicating the handling property forunloading the operation objects from the pallet, and based on the index,create or derive the control sequence including identification ordetermination of the operation object and/or the pallet loaded with theoperation object. Here, “handling property” represents the degree ofdifficulty associated with manipulation of the operation object by therobot. The handling property may quantitatively represent, as anexample, the degree of difficulty when the operation robot grasps theoperation object loaded on the pallet, the degree of difficulty when theoperation robot moves the grasped operation object (complication of theroute, or the like), and the like. Also, the “index” may be digitizedquantitatively and, for example, may represent a qualitativeranking/evaluation.

With this configuration, when there are plural candidates for operationobjects to be unloaded, it is possible to select a pallet loaded withoperation objects that may be easier to manipulate using the operationrobot based on the index indicating the handling property. Thisincreases the likelihood of the robot accurately grasp and manipulatethe targeted object, resulting in improvement of operating efficiencyand operating accuracy (success rate).

[6] In the above configuration, the data storage unit may storepositional information (layout information) about a plurality of theoperation objects on the pallet as two-dimensional information and/orthree-dimensional information for each operation object or for eachpallet.

With this configuration, positions of the operation objects placed onthe pallet can be identified for each operation object or for eachlayer. Also, the layout of the plural operation objects stacked on thepallet can be managed collectively as one-dimensional information orthree-dimensional information. Thus, for example, if one operationobject is specified, the other operation objects/SKUs loaded on the samepallet can be specified all at once. This makes it possible to improveoperating/processing efficiency, for example, by reducing resources(e.g., time) needed for inventory control and stocktaking.

[7] In the above configuration, the robotic system may further include asensor adapted to image the operation object or the pallet loaded withthe operation object, and the robot control unit may create or derivethe control sequence for loading the operation object on the pallet soas to allow the sensor to measure identification information (e.g., anidentification code or identification tag representing theidentification information) about the operation object.

With this configuration, even if an operation object is loaded on thepallet, the identification information about the operation object can bechecked via the sensor. For example, if the identification informationabout the operation object is attached at least to a side surface, theoperation object may be placed/stacked on the pallet with theidentification information exposed/visible. The identificationinformation attached to the operation object can be easily recognized bybeing imaged by a sensor such as a camera. Thus, even if an object is onthe pallet, the object can be identified and specified easily. In sodoing, when positional information about the plural operation objects onthe pallet are stored in advance (e.g., as two-dimensional informationor three-dimensional information as with [6] above), all operationobjects loaded on the pallet can be identified and located more easilyand simply all at once based on identifying and locating one operationobject. This makes it possible to further improve operating efficiencyin inventory control and stocktaking.

[8] In the above configuration, the robot control unit may indicate thatloading of the operation object on the pallet and/or unloading of theoperation object from the pallet are/is/will be complete, before thetask is actually complete. In some embodiments, the robot control unitmay set a status representing that loading/unloading of the operationobject is completed, before actual implementation of the task isfinished.

With this configuration, a completion status of a preceding stage may beset before the actual/real-world completion of such stage. Thus, a taskfor the next stage can be started smoothly on a timely basis, such as byusing the completion status of the preceding stage as a trigger forparallel execution or preparation, thereby enabling sophisticatedcooperation among units and reducing work delays.

[9] In the above configuration, the data deriving unit may deriveactually measured values or estimated values for the information aboutthe operation object and/or the pallet. Consequently, the position ofthe operation object on/outside the pallet, the position of the pallet,and the position of the identification information attached to theoperation object can be derived by actually measuring aspects of thecorresponding structures with a sensor (e.g., a camera) and/or byestimating via predetermined data (using, e.g., master data withoutactual measurement).

[10] A logistics system according to the present technology includes thecontroller according to any one of the above configurations; and arobotic system including the operation robot.

[11] In the above configuration, the controller may identify the roboticsystem and a region related to the robotic system and calculate tasks(including a unit task and a task made up of a combination of aplurality of unit tasks) based on the control sequence. The tasks mayinclude a task related to transportation of the pallet and/ormanipulation/transfer of the operation object by the robotic system. Therobotic system may execute a plurality of tasks across adjacent and/oroverlapping regions.

[12] A program according to the present technology causes a computer tofunction as the controller of any one of the above configurations.

[13] A recording medium according to the present technology is acomputer-readable non-transitory recording medium on which the aboveinstructions/configurations are stored.

[14] A control method according to the present technology may include acontrol method for a robotic system that includes an operation robotadapted to manipulate an operation object to load the operation objecton a pallet and/or unload the operation object from the pallet. Thecontrol method may be performed via a controller having a data derivingunit, a data storage unit, and a robot control unit. With this method,(1) the data deriving unit may derive first data including informationabout the operation object and information about the pallet beforeloading of the operation object on the pallet and/or unloading of theoperation object from the pallet; (2) the data storage unit may storethe first data; and (3) the robot control unit may create or derive acontrol sequence for loading the operation object on the pallet and/orunloading the operation object from the pallet based on the first databefore the loading of the operation object on the pallet and/orunloading of the handing object from the pallet, and based on thecontrol sequence, may instruct or control the operation robot to executea task of loading the operation object on the pallet and/or unloadingthe operation object from the pallet.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flowchart showing an example work sequence for arobotic system according to one embodiment of the present technology.

FIGS. 2A and 2B are perspective views illustrating example transportrobots according to one or more embodiments of the present technology.

FIG. 3 is a perspective view illustrating an example operation robotaccording to one or more embodiments of the present technology.

FIGS. 4A and 4B are perspective views illustrating an example operationrobot according to one or more embodiments of the present technology.

FIG. 5 is a block diagram showing an example of a hardware configurationand functional configuration of a robotic system according to oneembodiment of the present technology.

FIG. 6 is a front view (partial functional configuration diagram)showing an example of a flow of work performed by a robotic systemaccording to one embodiment of the present technology as well as anexample of a functional configuration of a controller provided on therobotic system.

FIG. 7 is a flowchart showing an example process flow for executing afirst task by operating a robotic system according to an embodiment ofthe present technology.

FIG. 8 is a flowchart showing an example of procedures for executing asecond task by operating a robotic system according to an embodiment ofthe present technology.

FIG. 9 is a schematic plan view showing an example environment in whicha robotic system according to one embodiment of the present technologycan operate.

DETAILED DESCRIPTION

The present technology provides a robotic system in which multiple units(e.g., various robots, various devices, a controller provided integrallytherewith or separately therefrom) are integrated in a sophisticatedmanner, a controller for the robotic system, a logistics system equippedtherewith, a method therefor, and the like. That is, a robotic systemaccording to an embodiment of the present technology is an integratedsystem capable of autonomously executing, for example, one or moretasks.

Here, “tasks” can include, for example, a robot and the like (1)accessing operation objects, (2) moving/transferring the operationobjects from one place to another and storing, (3) keeping, retrieving,and/or organizing the operation objects, (4) classifying the operationobjects into plural groups, and/or (5) imaging, observing, and managingthe operation objects.

Also, “tasks” can include, combinations of plural tasks executed duringwarehousing of operation objects (goods receipt, transportation, loading(palletizing), unloading (depalletizing), housing, etc.), replenishmentwith operation objects (unpacking, filling, replenishment, etc.), and/orshipping of operation objects (retrieval, pickup, packing, loading,palletizing, etc.) in a logistics system. Furthermore, “tasks” caninclude, for example, gripping or lifting operation objects at aspecified position, moving operation objects on a specified path, andreleasing, putting down, or placing operation objects at a specifiedposition in order to rearrange, regrip, or replace the operationobjects.

Furthermore, the robotic system and/or the controller according to thepresent technology can access operation objects, for example, via pluralunits such as a robot. Consequently, the robotic system can supportautomation of operations in a warehousing process, replenishmentprocess, shipping process, and the like of the operation objects.Furthermore, the robotic system and/or the controller according to thepresent technology can classify (group) operation objects as appropriate(e.g., based on the SKU differences), move or place the operationobjects to/at desired positions or specified positions, access theoperation objects, relocate or reclassify the operation objects in theplace or in other places, and/or rearrange or replace the operationobjects according to characteristics of the operation objects. In thiscase, the robotic system and/or the controller can read one or morepieces of identification information (e.g., a bar code or Quick Response(QR) code (registered trademark)) attached to one or more specificpositions or surfaces of the operation objects. The identificationinformation can be checked against master data as required, therebyidentifying and/or pinpointing the operation objects. Based on theidentification information, information associated with the operationobjects may be derived.

Furthermore, the robotic system according to the present technology canbe equipped with a sensor such as an imaging sensor to identifypositions and states (e.g., attitudes including orientations) ofoperation objects and/or a surrounding environment of the operationobjects. The imaging sensor can derive images of working positions(e.g., positions of the operation objects including a pick-up position,a drop position, and/or one or more positions along a path) for thetasks executed by individual units of the robotic system. Similarly, theimaging sensor can derive images of the operation objects at the workingpositions. Also, the robotic system (via, e.g., the controller) canprocess images of the operation objects in a predetermined order (e.g.,from top to bottom, from a lateral edge, or from an inner side of theoperation object). In so doing, states and situations of the images ofthe operation objects can be determined as appropriate by identifyingand classifying outer shapes and environments of the operation objectsfor example, based on colors of adjacent pixels, brightness, and/orchanges in those values on the images of the operation objects.

The robotic system can derive (via, e.g., the controller) and execute acontrol sequence for executing a task of accessing and manipulatingoperation objects, a task of transporting, moving, placing, storing, andotherwise manipulating the operation objects, and other tasks. Thecontrol sequence can include a combination of commands and/or settingsfor driving operative mechanisms (e.g., actuators) of the respectiveunits. The robotic system can create or derive a control sequence forexecuting various tasks, for example, by machine learning such as motionplanning or deep learning, or by another method.

Conventional robotic systems used in a typical logistics system, such asfor warehousing, replenishment, or shipping of operation objects, areoften unable to perform sophisticated interaction among plural units. Assuch, operator support is often required for successful transitionsacross different tasks. Also, conventional robotic systems often requirethe operator to classify ordered goods or place them in order.Furthermore, with the conventional robotic system, changing/replacingthe robotic units and/or corresponding operating procedures is oftendifficult. In particular, when the units require operator interventionand support, it is extremely difficult to autonomously or automaticallychange a control sequence. Moreover, conventionally, in the case ofloading operation objects on a pallet or the like, it is difficult toautomate loading of different SKUs in a mixed manner on the same pallet.As such, mixed loading conventionally requires operator intervention.This hinders full-automation of a logistics system.

Also, the conventional robotic system executes a task of gripping anoperation object at one position, moving to another position in thisstate, and releasing the operation object. However, conventional systemsoften lack the capacity to account for storage efficiency for objectsand operating efficiency of each unit. Accordingly, conventional systemsoften require operator intervention and support (adjustment, redo,supplementation, system stop, etc.) to achieve the desiredstorage/operating efficiency.

In contrast, the robotic systems/controllers according to variousaspects of the present technology (described below by example) canadjust and control interaction among separate units (e.g., the operationrobot and transport robot) in executing tasks and promote cooperationamong different units. Consequently, the operator intervention andsupport is reduced/eliminated, and the storage efficiency for operationobjects, operating efficiency, and economy can be improved. Furthermore,by acquiring and managing information about the operation objects andthe pallet at appropriate timing for palletizing and depalletizing ofthe operation objects, it is possible automate loading of the operationobjects having different SKUs on the same pallet in a mixed manner.

Also, the robotic system/controller can reliably identify an operatingrange, a motion path, positions and states of operation objects, or acombination thereof concerning each unit, and execute tasks smoothlyacross different units. In so doing, the robotic system/controller canfurther increase efficiency of space usage based on identifying andprocessing shape information, identification information, positionalinformation, and other similar information about the operation objects.The robotic system/controller can create or derive, and execute one ormore algorithms configured to place tasks of different units in order.The controller can further establish one or more protocols forcontrolling interaction among separate units and derive a suitablecontrol sequence that can implement sophisticated cooperative motionsamong various units based on information about the states of operationobjects.

Embodiments according to an example of the present technology will bedescribed below with reference to the accompanying drawings. However,the embodiments described below are merely illustrative, and variousmodifications and technical applications not mentioned hereinafter arenot intended to be excluded. That is, the example of the presenttechnology can be implemented in various modified forms withoutdeparting from the spirit and scope of the present technology. Also, inthe following drawings, the drawings are schematic and do notnecessarily represent actual sizes, ratios or the like. Furthermore, thedrawings may not be shown in their true size relations. Also, theembodiments described below are part of the embodiments of the presenttechnology rather than all the embodiments of the present technology.Furthermore, any other embodiment obtained by those skilled in the artbased on embodiments of the present technology without the need forcreative activities is included in the scope of protection of thepresent technology.

Also, in each embodiment, technique introduced herein can be carried outwithout particular details of the technique. Furthermore, detaileddescription of well-known functions such as specific functions orroutines is omitted to avoid making the present technology unnecessarilydifficult to understand. Besides, detailed description of well-knownstructures or processes often associated with robotic systems andsubsystems may also be omitted for the sake of clarification. Referenceto “an embodiment,” “one embodiment,” or the like herein means that aspecific feature, structure, material, or characteristic is included inat least one embodiment of the present technology. Therefore, themention of such a phrase herein does not necessarily refers to the sameembodiment. On the other hand, such references are not always mutuallyexclusive. Furthermore, a specific feature, structure, material, andcharacteristic can be combined in any appropriate manner in one or moreembodiment. It should be understood that various illustrated embodimentsare presented only in an explanatory manner and are not necessarilyshown in their true scales.

Also, many embodiments or aspects of the present technology includeprocesses, steps, routines, blocks, and the like executed by aprogrammable computer or controller (controller) and take the form ofcommands executable by the computer or controller. It should beunderstood by those skilled in the art that the disclosed technique canbe implemented by a system of computers or controller other than thoseshown in the embodiments. The technique described herein can be carriedout in a special-purpose computer or data processor programmed,configured, or built to execute one or more computer-executable commandsdescribed below.

Thus, the terms “computer” and “controller” generally used hereininclude any data processor as well as an Internet-enabled apparatus anda hand-held device (including palmtop computers, wearable computers,cellular or mobile phones, multi-processor systems, processor-based orprogrammable household electric appliances, network computers, andminicomputers). Information handled by the computer and controller canbe provided to any appropriate display media including, for example, aliquid crystal display (LCD).

Commands issued to execute tasks executable by the computer or thecontroller can be stored in hardware, firmware, or an appropriatecomputer-readable non-transitory recording medium including acombination of hardware and firmware. Also, each command can beincluded, for example, in any appropriate memory device including aflash drive and/or other appropriate medium.

“Coupled,” “connected,” and other similar terms may be used to describestructural relationships among components. However, it is not intendedthat these terms are interchangeable. Specifically, in specificembodiments, the term “connected” can be used to indicate that two ormore elements are in direct contact with one another. Unless otherwiseunderstood clearly from the context, the term “coupled” can be used toindicate that two or more elements are in contact with one anothereither directly or indirectly (by involving intervening elements amongthem), indicate that two or more elements collaborate or interact oneanother (such as when the elements are in a cause-and-effectrelationship as in the case of signal transmission/reception or afunction call), or indicate both states.

Application Example

FIG. 1 is a schematic flowchart showing an example work sequence of alogistics system including a robotic system according to one embodimentof the present technology. In some embodiments, the work sequence mayinclude a warehousing process P10 of receiving operation objects (e.g.,containers such as carton boxes containing items, according to thepresent embodiment) into a distribution center or the like. The worksequence may also include a replenishment process P20 of replenishing astorage space (e.g., a shelf or the like) with operation objects, whichmay be placed, for example, on a pallet or the like. The work sequencemay further include a shipping process P30 of picking up and shippingordered items from the stored/replenished operation objects.

At block S101, operation objects carried in by a vehicle or the like maybe received. In receiving, the objects may be unloaded using, forexample, a devanning robot. At block S102, using, for example, a sortingsystem, the operation objects may be transported on a conveyor or thelike to a waiting position of reloading pallets. At block S103, theoperation objects that have been transported may be loaded, for example,onto an appropriate pallet by a palletizing robot.

At block S104, the pallet with the operation objects placed thereon maybe transported to a predetermined position. The pallet may betransported using, for example, a pallet automated guided vehicle (AGV)configured to move pallets by carrying or lifting them. Thepredetermined position may be a location where operation objects areunloaded from the pallet, for example, by a depalletizing robot. Atblock S105, the operation objects may be transported by a conveyor orthe like to a waiting position of storage shelves. During transport, theoperation objects may be exposed/unpacked by cutting or opening the topof the operation objects, using, for example, a top-opening machine. Atblock S106, the operation objects may be stored, such as in anappropriate position of the shelf (via, e.g., a fetching robot) and/ortransported to a shelf storage location (via, e.g., a shelf AGV, andkept in an appropriate state.

In some embodiments, the shipping process P30 may correspond to an orderfor items). At block S107, a shelf having operation objects thereon withthe ordered items may be transported to a retrieval position using, forexample, a shelf AGV. At the retrieval position, a desired operationobject (e.g., a box or a container) may be retrieved from the shelf by,for example, a fetching robot. At block S108, using, for example, apiece-picking robot, the ordered items may be picked from the operationobject, and transported to a shipment container or the like. At blockS109, the shipment container may be packed using, for example, a packingmachine. At block S110, the packed shipment container may be loadedinto, for example, an appropriate cart or vehicle and shipped.

In this way, the robotic system can include or interact with operationrobots (e.g., transporting/sorting units such as a devanning robot,palletizing robot, depalletizing robot, fetching robot, piece-pickingrobot, packing machine, and the like) configured to move operationobjects between different locations. Also, the robotic system accordingto one embodiment can include a sorting system, pallet AGV, and shelfAGV as transport robots serving as transport units.

Here, FIGS. 2A and 2B are perspective views illustrating exampletransport robots 11 and 12 according to one or more embodiments. Thetransport robots 11 and 12 can include a mobile/wheeled robot such as ashelf AGV configured to transport pallets between predeterminedpositions. For example, the transport robots 11 and 12 have such outsidedimensions as to allow the transport robots 11 and 12 to move underpallets and/or between pallets. Also, the transport robots 11 and 12 caninclude a lifting mechanism (e.g., a folk) configured to lift palletsfrom the ground (e.g., a transport path surface).

The transport robots 11 and 12 can be navigated using variousmechanisms. For example, the transport robots 11 and 12 can travel bythemselves tracing a predetermined path provided (e.g., a floor marking,such as painting or tape). The transport robots 11 and 12 can travelbased on instructions from a controller of a robotic system according toan embodiment. Also, the transport robots 11 and 12 can calculatecurrent positions via a mapping/positioning mechanism (e.g., a deadreckoning system, a system based on laser, and/or a system based on aradio communications signal), and move along a specified path and routebased on the current positions.

Note that a robotic system according to an embodiment can transmit(e.g., via a controller as a stand-alone device or part of another unit)a target position of a pallet to be transported, a holding position ofthe pallet, identification information about the pallet, a path, amotion plan, or a combination thereof to the transport robots 11 and 12.Based on communication instruction information, the transport robots 11and 12 can execute a task of, for example, moving to the holdingposition of the pallet to be transported, lifting the pallet,transporting the pallet to a specified position, and/or placing thepallet at the specified position. Furthermore, the transport robots 11and 12 can execute or complete the task by returning the pallet to theoriginal holding position or a different storage location.

FIGS. 3, 4A, and 4B are perspective views illustrating example operationrobots 13 and 14 according to one or more embodiments of the presenttechnology. The operation robots 13 and 14 can include, for example, arobot configured to transfer operation objects 16 (see FIG. 4) betweenpredetermined positions. For example, the operation robots 13 and 14have structural members including end effectors 131 and 141 and robotarms 132 and 142 equipped with the end effectors 131 and 141 at thetips. The end effectors 131 and 141 may be grippers capable of grippingoperation objects by decompression suction or vacuum suction. Also, theend effectors 131 and 141 may be a hand type gripper for picking. Thisallows the operation robots 13 and 14 to grip the operation objects fromabove and/or along a lateral direction. Also, the operation robots 13and 14 may fixed to a predetermined location or may be configured forlocomotion. In addition, the end effector 141 of the operation robot 14may be equipped with a plate guide or the like configured to support(hold) the operation object 16 from below (e.g., devanning gripper) andsuitably functions as a devanning robot or a fetching robot.

The operation robots 13 and 14 can be driven, for example, oninstructions from a controller of a robotic system according to anembodiment. Also, the operation robots 13 and 14 can calculate currentpositions of operation objects and manipulate the operation objectsalong specified paths and routes based on the current positions. Therobotic system can transmit (e.g., via a stand-alone controller or adevice that is part of another unit) target positions and pickuppositions of the operation objects, identification information about theoperation objects, paths, motion plans, or a combination thereof to theoperation robots 13 and 14. Based on communication instructioninformation, each of the operation robots 13 and 14 can execute a tasksuch as moving the end effectors 131 and 141 to a grip position of anoperation object, gripping and lifting the operation object,transporting the operation object to a specified position, and/orplacing the operation object at the specified position.

Configuration Example

FIG. 5 is a block diagram showing an example of a hardware configurationand functional configuration of a robotic system according to oneembodiment of the present technology. For example, a robotic system 500can be equipped with electronic/electrical equipment such as a processor502, storage device 504, communications device 506, input-output device508, actuation device 512, transport motor 514, sensor 516 orcombination thereof. The robotic system 500 can be equipped with morethan one of these elements and devices.

Also, these circuits and/or devices can be coupled together via wiredconnections and/or wireless connections. For example, the robotic system500 can include buses such as a system bus, a Peripheral ComponentInterconnect (PCI) bus or a PCI Express bus, a HyperTransport orIndustry Standard Architecture (ISA) bus, Small Computer SystemsInterface (SCSI) bus, Universal Serial Bus (USB), an IIC (I2C) bus, andan IEEE 1394 (Institute of Electrical and Electronic Engineers) bus(also referred to as a “Firewire” bus).

Also, to provide wired connections among devices, the robotic system 500can include, for example, a bridge, an adapter, a controller, and/oranother signal-related device. Wireless connections may be based, forexample, on a cellular communications protocol (e.g., 3G, 4G, LTE, or5G), a wireless local area network (LAN) protocol (e.g., wirelessfidelity (WIFI)), a peer-to-peer or device-to-device communicationsprotocol (e.g., Bluetooth (registered trademark) or Near-FieldCommunications (NFC)), an Internet of Things (IoT) protocol (e.g.,NB-IoT or LTE-M), and/or another wireless communications protocol.

The processor 502 can include a data processor (e.g., central processingunit (CPU), special-purpose computer, and/or on-board server) configuredto execute commands (e.g., software instructions) stored in the storagedevice 504 (e.g., a computer memory). The processor 502 can executeprogram instructions so as to control, or interact with, other devices,thereby making the robotic system 500 perform various motions,operations, and/or manipulations, in a task.

The storage device 504 can include a computer-readable non-transitoryrecording medium with program instructions (e.g., software) storedthereon. Examples of the storage device 504 include a volatile memory(e.g., cash and/or random access memory (RAM)), a non-volatile memory(e.g., flash memory and/or magnetic disk drive), a portable memorydrive, and a cloud storage device.

In some embodiments, the storage device 504 further stores processingresults, predetermined data, a predetermined threshold, a predeterminedparameter, and allows access to them as appropriate. For example, thestorage device 504 can store master data 552 including informationassociated with operation objects that can be processed and/ormanipulated by the robotic system 500.

The master data 552 can include dimensions, shapes (e.g.,computer-generated templates of possible states and computer-generatedmodels used to recognize the operation objects in different states),colors, images, identification information (e.g., bar codes, QuickResponse (QR) codes (registered trademark), and logos as well aspositions where the codes and the like are expected to be provided),expected weights, or a combination thereof regarding the operationobjects. Also, the master data 552 can include information associatedwith maneuvering of the objects, such as respective center-of-mass(center-of-gravity) positions of the operation objects, expectedmeasurement values (e.g., various measurement of force, torque,pressure, and/or a contact reference value) of sensors in response toone or more motions/movements, or a combination thereof.

Also, for example, the storage device 504 can store object tracking data554 that is produced when the operation objects are tracked and/ormanipulated by the robotic system 500. The object tracking data 554 caninclude a log of scanned or manipulated operation objects. Also, theobject tracking data 554 can include image data (e.g., photos, pointclouds, and live videos that are provided) of operation objects at oneor more positions (e.g., a specified pickup position or drop position,and/or, a position on a conveyor belt) and/or positions/states(orientations, etc.) of the operation objects at one or more positions.

The communications device 506 can include a circuit configured tocommunicate with an external device or remote device via a network. Forexample, the communications device 506 can include a receiver, atransmitter, a modulator/demodulator (modem), a signal detector, asignal encoder/decoder, a connector port, and/or a network card. Thecommunications device 506 can be configured to receive, transmit, and/orprocess signals according to one or more communications protocols (e.g.,the Internet Protocol (IP) and/or a wireless communications protocol).Using the communications device 506, the robotic system 500 can exchangeinformation among the units of the robotic system 500 and/or withsystems/apparatus external to the robotic system 500 for example, forthe purposes of data collection, analysis, reporting, trouble shooting,and the like.

The input-output device 508 can include a user interface deviceconfigured to exchange information with the operator and/or receiveinformation from the operator. For example, to exchange information withthe operator, the input-output device 508 can include a display 510and/or other output device such as a speaker, a tactile circuit, and/ora tactile feedback device. Also, the input-output device 508 can includecontrol and receiving devices such as a keyboard, a mouse, atouchscreen, a microphone, a user interface (UI) sensor (e.g., a cameraused to receive motion commands), and/or a wearable input device. Usingthe input-output device 508, the robotic system 500 can interact withthe operator when performing tasks, motions, operations, manipulations,or a combination thereof.

A robotic unit associated with the robotic system 500 can include astructural member that is connected by a joint for motions (e.g.,rotation and/or translation) and an end effector. The end effector canbe configured, for example, to pick up, grip, spin, and/or weld withrespect to a target object. An example of the structural member caninclude a robot arm and the like configured to maneuver the end effector(e.g., the robot arm 132 and 142 of FIGS. 3, 4A, and 4B). Also, at oraround a joint, the robotic unit can include an actuation device 512(e.g., a motor, an actuator, a wire, an artificial muscle, and/or anelectroactive polymer) configured to drive or maneuver (e.g., displaceand/or orient) the structural member. The robotic unit may also includethe transport motor 514 configured to transport the unit from oneposition to another.

The robotic system 500 can include a sensor 516 to obtain informationused in driving or maneuvering the structural member and/or performingtransport work of various units. The sensor 516 can include variousdevices configured to detect or measure one or more physical properties(e.g., states, conditions, positions, and the like of one or more jointsor structural members) of the robotic system 500 and/or characteristicsof a surrounding environment. Some examples of the sensor 516 caninclude an imaging sensor 522, position sensor 524, and contact sensor526 as well as an accelerometer, a gyroscope, a force sensor, a straingage, and/or a torque sensor.

The imaging sensors 522 can include, for example, a visible cameraand/or infrared camera, a two-dimensional imaging camera and/orthree-dimensional imaging camera (2D vision and/or 3D vision), and aranging device such as a lidar or radar configured to detect thesurrounding environment. In applying automated inspection, robotguidance, or another robot, the imaging sensor 522 can generate datasuch as digital images and/or distance measurements (e.g., point clouds)that may be used to control various units and the controller.

To manipulate each operation object, the robotic system 500 can identifypositions of various elements by analyzing images of specified regions(e.g., regions containing a grip position, a pick-up position, a dropposition, and other working positions regarding the operation object).For example, the imaging sensor 522 can include the above-mentionedcameras and ranging devices configured to generate image data anddistance data of the specified regions. Based on the images and/or theranging data captured by the imaging sensor 522, the robotic system 500can determine, for example, the grip position, pick-up position, dropposition, and/or other working positions regarding the operation object.Note that the task can include scanning the operation object to keep alog of the operation object during transportation/acceptance. In thiscase, the imaging sensor 522 can include one or more scanners (e.g., abar code scanner and/or QR code scanner (registered trademark))configured to scan identification information about the operation objectduring transportation of the operation object.

Also, the position sensor 524 can include a position encoder and/or apotentiometer configured to detect the position of a joint and/or astructural member in a corresponding robot unit of the robotic system500. While performing the task, the robotic system 500 can use theposition sensor 524 to track the positions and/or states (orientations,etc.) of the structural members and/or joints.

The contact sensor 526 can include, for example, a pressure sensor, aforce sensor, a strain gage, a piezoresistive/piezoelectric sensor, acapacitive sensor, an elastic resistance sensor, and/or other tactilesensors configured to measure properties related to direct contactbetween physical structures or surfaces. In some embodiments, thecontact sensor 526 can measure a property corresponding to a gripoperation of the end effector with respect to an operation object.Therefore, the contact sensor 526 may be configured to detect a physicalquantity that represents contact or an amount of grip between theoperation object and gripper and can output a quantified measurementvalue (e.g., force, torque, pressure, and/or a contact reference value).Note that the contact reference value may include one or more readoutsof force or torque in relation to force applied to the operation objectby the end effector.

Note that although the robotic system 500 is described with respect to awarehouse and logistics system as examples, the robotic system 500 canbe configured to execute various tasks in other environments or forother purposes, such as to carry out manufacture, assembly, packing,health care and/or other types of automated operation. Also, the roboticsystem 500 can include other units such as non-illustrated manipulators,service robots, and/or modular robots. Further, the robotic system 500can include, for example, various types of unloading/loading robotsadapted to operate in a warehouse or distribution/transportation hub.Different types of unloading/loading robots may be used to transportoperation objects from various types of locations, such as from a cagecart or pallet to a conveyor or another pallet, a sorting system, anunpacking robot used to unpack operation objects, a top-opening machine,a container switching robot used to transport operation objects from onecontainer to another, a packing robot used to pack operation objects, apacking machine, or a combination thereof.

Operation Example

FIG. 6 is a front view (partial functional configuration diagram)showing an example work flow of a robotic system according to oneembodiment of the present technology. FIG. 6 further illustrates anexample functional configuration of a controller provided on the roboticsystem. A robotic system 600 according to the present technology may beimplemented, for example, in a warehouse or the like, which is afoothold for warehousing, replenishment, storage, and shipping in alogistics system. The robotic system 600 can include a controller 610 asan apparatus adapted to adjust and control operation of various units ofthe robotic system 600. The robotic system 600 can also be a systemparticularly suitable for loading of operation objects on a pallet andunloading of operation objects from the pallet in the warehousingprocess P10, in the warehousing process shown in FIG. 1.

As described above, the controller 610 may be configured as astand-alone device or part of another unit. The controller 610 canadjust and control motions in tasks to be executed by units such as thetransport robots 11 and 12 (the transport robot 11 is shown in FIG. 6 asan example; the same applies to the following) and the operation robots13 and 14 (the operation robot 13 having a hand-type end effector forpicking, instead of a gripper, is shown in FIG. 6; the same applies tothe following), as well as units such as other robots and transportmeans. More specifically, the controller 610 can be communicativelycoupled to the transport robot 11, the operation robot 13, and thesensor 516 including the imaging sensor 522. The controller 610 may becoupled to other non-illustrated systems or devices, such as warehousemanagement system (WMS), other host system, and/or external system, asneeded.

The controller 610 may include a data deriving unit 612, a data storageunit 614, and a robot control unit 616. In some embodiments, thecontroller 610 may include the processor 502, the storage device 504,and/or the communications device 506 illustrated in FIG. 5. Inparticular, the processor 502 can function as the data deriving unit 612and robot control unit 616 while the storage device 504 can function asthe data storage unit 614. The robotic system 600 can be configured toexecute a first task of loading operation objects 16 in a pallet 15and/or a second task of unloading the operation objects 16 from thepallet 15.

First Task: Loading Operation Objects 16 in Pallet 15

In the first task, tasks A1 to A5 shown below can be executed incombination in an order according to corresponding timing.

A1: The pallet 15 kept in a storage area 101 may be transported from itsstorage location to an access position 601 (stop location SL) in agrouping area 103 by the transport robot 11.

A2: Each operation object 16 temporarily placed, for example, at atransport (output) location 602 (transport location OL) on a conveyor inthe grouping area 103 may be gripped and picked up by the operationrobot 13.

A3: The gripped operation object 16 may be moved from the transportlocation 602 to the access position 601.

A4: The operation object 16 may be loaded in a specified vacant positionon the pallet 15 held at rest by the transport robot 11.

A5: The pallet 15 loaded with the operation objects 16 may be returnedfrom the access position 601 to the storage location in the storage area101 by the transport robot 11.

Note that in the present embodiment, the described example includesholding and moving the pallet 15 by the transport robot 11. However, thepresent embodiment is not limited thereto. For example, loading andunloading of the operation objects 16 may be performed in a state wherethe pallet 15 is placed on the ground, and/or the pallet 15 may be movedby a transport means (forklift or the like) other than the transportrobot 11. Moreover, loading and unloading of the operation objects 16may be performed in a state where the pallet 15 is placed on a conveyer.Furthermore, the pallet 15 may be transported to a predetermined pointalong the transport path defined by the conveyer and finally moved fromthe conveyer by another transport means (e.g., forklift or the like).

FIG. 7 is a flowchart showing an example process flow for executing thefirst task by operating a robotic system according to an embodiment ofthe present technology. First, in block 701, the data deriving unit 612may derive first data. The first data may include information aboutoperation objects 16 before executing the task A1 and/or informationabout each pallet 15. The data storage unit 614 may store the first databy associating the first data with the operation objects 16 and pallets15.

For example, the data deriving unit 612 can obtain an image of theoperation object 16 temporarily placed at the transport location 602 inthe grouping area 103, via a 3D-vision or other imaging sensor 522. Thedata deriving unit 612 may process the obtained image to deriveinformation about the operation object 16, such as identification,position, shape, and/or estimates of the weight and/or center of mass.The data deriving unit 612 may further derive the information about theoperation object 16 based on the master data 552 as required.Alternatively or additionally, information about the operation object 16and pallet 15 in the master data 552 can be used to estimate or pinpointinformation, status, or state about the operation objects 16 and pallets15. Note that as a technique for identifying/tracking an operationobject 16, for example, methods described in Japanese PatentApplications Nos. 2019-118678 and 2019-080213, U.S. patent applicationSer. No. 16/258,120, and other applications can be used. Alternativelyor additionally, the data deriving unit 612 can derive information aboutthe operation object 16 from a host system such as the WMS. Theinformation about the operation object 16 can be derived in advancebefore the operation object 16 is placed at the transport location 602.

In block 702, based on the information about the operation objects 16and information about each pallet 15, the robot control unit 616 maydetermine pallet information representing one of the pallets 15 on whichthe operation objects 16 are to be loaded. The robotic control unit 616can select the targeted pallet based on the information about theoperation objects 16 and/or the available pallets. Examples of theinformation about the operation objects 16 may include SKU, size, shape,weight, quantity, and the like. Examples of the information about theavailable pallets may include size, shape, available area, loadingcapacity, quantity, surface location and/or height position ofpreviously loaded or existing operation objects on the potentialpallets. The robot control unit 616 can be configured to process theinformation to select a candidate for the pallet 15 that increases ormaximizes the storage efficiency of the entire warehouse and/orminimizes the occupied spaces of or between the operation objects 16 onthe SKU basis.

The operation objects 16 having the same or similar SKUs on the pallet15 may be arranged in a concentrated manner (e.g., by minimizing theoccupied spaces of or between the objects). In some embodiments, theoperation objects 16 having the same or similar SKUs may be loaded onthe pallet 15 for un-mixed loading. The robotic system may also load aplurality of types of the operation objects 16 having different SKUs ona pallet for mixed loading. Loading types and/or objects may be selectedbased on a category or a state of the pallet 15. For example, in thecase of temporarily loading the operation objects 16 on the pallet 15,the controller 610 can select the operation objects 16 to be loaded onthe pallet 15 with a choice of non-mixed loading or mixed loading asappropriate. The controller 610 may further derive the loading order ofthe respective operation objects 16 as appropriate. Inselecting/deriving the operation objects 16, the loading type, and/orthe loading order, the controller 610 may process or consideroperability of the operation objects 16 after the temporal loading(e.g., accessibility in unloading the ordered goods, and the like) inaddition to the respective types of information described above.Moreover, it is also possible to select one or more optimum pallets 15while considering the transport path to the grouping area 103 of thepallet 15 loaded with the operation objects 16.

Also, the controller 610 can select the operation objects 16 and loadthem on the pallet 15 based on, for example, the information about thepallet 15 (in particular, available area, plane, height position, andthe like), regardless of the loading type (e.g., temporary loading,un-mixed loading, and/or mixed loading). Furthermore, the controller 610may control the ratio between un-mixed loading and mixed loading. Thecontroller 610 can adjust the degree of mixed loading on each pallet 15based on information about (condition for) the operation objects 16. Therobotic system may further arrange multiple pallets 15 on which theoperation objects 16 can be loaded. The arrangement of the pallets 15and loading operation may be performed simultaneously and/or inparallel. Moreover, the controller 610 may select a pallet 15 of morepreferable condition to thereby further improve the working efficiency.Furthermore, the robotic system may temporarily place the operationobjects 16 on another intermediate placing means or a temporary placinglocation (i.e., not the final targeted destination of the task) beforeloading the operation objects 16 on the pallet 15 (i.e., the finaltargeted destination of the task).

In block 703, the robot control unit 616 can specify the transport robot11 used to transport the determined one or more pallets 15 from thestorage area 101 to the access position 601 in the grouping area 103.The controller 610 can create or derive a control sequence, for example,by machine learning or the like. The control sequence may include atransport path of the pallet 15, such as the path intended for thetransport robot 11. Based on the control sequence, the robot controlunit 616 can instruct the specified transport robot 11 to execute thetask A1 of transporting the pallet 15 to the access position 601.

In block 704, based on information about the operation objects 16, therobot control unit 616 may create or derive a control sequence, forexample, by machine learning or the like either. The robotic controlunit 616 may create or derive the control sequence simultaneously with,before, or after implementation of the task A1. The control sequence maybe for picking up, gripping, and/or transferring the operation object 16from the transport location 602 to the access position 601 using theoperation robot 13. Based on the control sequence, the robot controlunit 616 may instruct and control the operation robot 13 to execute thetasks A2 and A3 of gripping. Accordingly, the robotic system may movethe operation object 16 to the access position 601.

The timing to execute the tasks A2 and A3 may not be specificallylimited. For example, the robot control unit 616 may instruct theoperation robot 13 to perform at least a part of the control sequencebefore the pallet 15 reaches the access position 601. In that case,before the pallet 15 reaches the access position 601, the operationrobot 13 may grip and move the operation objects 16 to the accessposition 601 and immediately upon arrival of the pallet 15. Accordingly,the operation robot 13 can execute the task A4 of loading the operationobjects 16 in the pallet 15.

In block 705, the controller 610 can obtain one or more images of theoperation objects 16 and/or pallet 15 at the access position 601, suchas via a 3D-vision or other imaging sensor 522. The controller 610 canprocess the resulting image data, and thereby determine the states andsituations of the operation objects 16 and/or pallet 15. Furthermore, inblock 705, the robot control unit 616 may specify positions for theoperation objects on the pallet 15 based on information about theoperation objects 16 and information about the pallet 15. Some examplesof the processed information regarding the pallet 15 may include loadingstate and concentration state (e.g., density) of the operation objects16 and/or available area on the pallet 15. Accordingly, the roboticcontrol unit 616 may create or derive a control sequence, for example,by machine learning or the like. The control sequence may include travelpaths of the operation objects 16 to the corresponding positionsdescribed above. Based on the control sequence, the robot control unit616 can instruct the operation robot 13 to execute the task A4 ofdropping the operation objects 16 at the specified positions on thepallet 15, which may held by the transport robot 11.

The robot control unit 616 can designate, for example, any position onthe pallet 15 as reference coordinates for the operation robot 13 toaccess the pallet 15. The rest position of the pallet 15 can becalculated based on the access position 601 at which the transport robot11 holding the pallet 15 stops. The position of the pallet 15 on thetransport robot 11 may deviate from the standard position. Thecontroller 610 can calculate positions of the transport robot 11 andpallet 15 from image data of the pallet 15 at the access position 601,and the data deriving unit 612 can derive relative positionalinformation about the transport robot 11 and pallet 15. Then, suitably,the robot control unit 616 can correct the positional information aboutthe pallet 15 at the access position 601 based on the relativepositional information about the two. The robot control unit 616 cancreate or derive a control sequence for the task A4, for example, bymachine learning or the like, based on the corrected positionalinformation about the pallet 15.

Once the operation objects 16 is on the pallet 15, in block 706, therobot control unit 616 can create or derive a control sequence, forexample, by machine learning or the like. The control sequence mayinclude a transport path along which the pallet 15 loaded with theoperation objects 16 may be returned from the access position 601 to thestorage location in the storage area 101 via the transport robot 11.Based on the control sequence, the robot control unit 616 can instructthe transport robot 11 to execute the task A5 of transporting andreturning the pallet 15 to the storage area 101.

When the above-mentioned first task of loading the operation objects 16on the pallet 15 is completed, in block 707, the data deriving unit 612can derive second data including information about the pallet 15 loadedwith the operation objects 16 and information about the operationobjects 16 on the pallet 15. The second data can include informationabout the loaded locations of the operation objects 16 on the pallet 15and information about SKU thereof. The data storage unit 614 can storethe second data in association with the operation objects 16 and thepallet 15.

While loading, the robot control unit 616 can create or derive a controlsequence for concentrating the operation objects 16 (i.e., reducingoverall space occupied by or between the objects) based on the seconddata. In other words, the robot control unit 616 can create or derivethe control sequence such that the operation objects 16 will be placedin a concentrated manner on the pallet 15 such that unused or unoccupiedspace will be minimized. Based on the properties of the operationobjects 16, the robot control unit 616 can create or derive, forexample, a control sequence such that the operation objects 16 havingthe same or similar SKUs are physically grouped together (i.e., locatedwithout other objects between the similar SKUs). Also, the roboticcontrol unit 616 can create or derive the control sequence forphysically grouping the operation objects 16 having equal or similar inshape and/or size in a concentrated manner. The second data derived inthis way may be updated for each operation object 16 and/or for eachloading operation performed on the operation object 16. The derivedsecond data can be used as appropriate in the second task (e.g.,unloading of operation objects 16 from the pallet 15) described later.

Furthermore, based on the second data, the data storage unit 614 canstore positional information (e.g., layout information) about one ormore operation objects 16 on the pallet 15. The stored positionalinformation can include two-dimensional representation for eachoperation object 16 and/or for each layer of the pallet 15. The storedpositional information can further include three-dimensionalrepresentation of the entire pallet 15.

Also, each operation object 16 may have an identification code or anidentification tag on, for example, on or about a surface thereof asidentification information. The identification information about theoperation object 16 can be exposed and be visible from the outside ofthe pallet 15 while the operation object 16 is stacked on the pallet 15.Then, by checking the identification information about the operationobject 16 using the imaging sensor 522 or the like, even if theoperation object 16 remains loaded on the pallet 15, the operationobject 16 can be identified without manipulating the operation object16. When the positional information about the operation objects 16 onthe pallet 15 is stored as two-dimensional information orthree-dimensional information, identification or determination of one ofthe stored operation objects 16 can directly lead toidentification/determination of the remaining operation objects 16stored on the pallet 15. Accordingly, the operation objects of eachlayer on the pallet 15 or each pallet 15 may be located based onidentifying/locating one object thereon. Note that the imaging sensor522 can be configured to image the pallet 15 when the transport robot 11holding the pallet 15 is at rest and/or when the transport robot 11holding the pallet 15 is moving.

Also, the robot control unit 616 may be configured to determine orindicate that loading of an operation object 16 on the pallet 15 iscompleted before the loading work (task A4) of the operation object 16on the pallet 15 is actually finished. For example, by checking anoperating state of an end effector of the operation robot 13 or bykeeping track of movement of the operation object 16 using the imagingsensor 522, it can be determined that loading of the operation object 16is completed. The robotic control unit 616 can determine or indicateearlier finished status when the entire operation object 16 or most ofthe operation object 16 is moved onto the pallet 15, when part of theoperation object 16 touches the pallet 15.

The robotic system 600 can execute the first task (loading of operationobjects 16 on the pallet 15) and the second task (unloading of operationobjects 16 from the pallet 15) made up of below-mentioned unit tasks B1to B5 to be executed in combination in appropriate order withappropriate timing.

Second Task: Unloading of Operation Objects 16 from Pallet 15

The second task can include unit tasks B1 to B5 shown below and executedin combination in appropriate order with appropriate timing.

B1: The pallet 15 kept in the storage area 101 and loaded with theoperation objects 16 may be transported from the storage location to theaccess position 601 (stop location SL) in the grouping area 103. Thepallet 15 may be transported by the transport robot 11.

B2: Each operation object 16 on the pallet 15 may be gripped and pickedup by the operation robot 13.

B3: The gripped operation object 16 can be moved from the accessposition 601 to the transport location 602.

B4: The operation object 16 can be dropped and temporarily placed at thetransport location 602.

B5: The pallet 15 that previously held the moved/unloaded operationobjects 16 can be returned from the access position 601 to the storagelocation in the storage area 101. The pallet 15 can be returned by thetransport robot 11.

Similar to the description of the first task above, the second task isdescribed with the pallet 15 being held and moved by the transport robot11. However, the present embodiment is not limited thereto.

FIG. 8 is a flowchart showing an example of procedures for executing thesecond task by operating a robotic system according to an embodiment ofthe present technology. In block 801, the data deriving unit 612derives, for example, first data including information about operationobjects 16 before executing the task B1. The data deriving unit 612 canalso derive information about each pallet 15. The data storage unit 614can store the first data by associating the first data with theoperation objects 16 and pallets 15.

For example, based on the second data stored in the data storage unit614 and/or the master data 552, the data deriving unit 612 can deriveinformation about the operation objects 16 such as identificationinformation, positional information, shape information, and/orinformation on the weight and center of mass. The data deriving unit 612can further determine identification information and/or storage locationinformation about the pallet 15 on which the operation objects 16 areloaded. In block 802, based on the information about the operationobjects 16 and information about the pallets 15 kept in the storage area101, the robot control unit 616 can select a pallet 15 from which theoperation objects 16 are to be unloaded. The robot control unit 616 candetermine one or more optimal pallets 15 according to a transport pathand the like to the grouping area 103. In one embodiment, based on theabove first and second data, the robot control unit 616 can determinepallet information representing the one of the pallets 15 from which theoperation objects 16 are to be unloaded.

In block 803, the robot control unit 616 can specify the transport robot11 used to transport the determined one or more pallets 15 from thestorage area 101 to the access position 601 in the grouping area 103.The robot control unit 616 can create or derive a control sequence, forexample, by machine learning or the like. The control sequence caninclude a transport path of the pallet 15 transported by the transportrobot 11. Based on the control sequence, the robot control unit 616 caninstruct the specified transport robot 11 to execute the task B1 oftransporting the pallet 15 to the access position 601.

When an end effector of the operation robot 13 is not at the accessposition 601, simultaneously with, before, or after the task B1, therobot control unit 616 can instruct the operation robot 13 to executethe task of moving the end effector of the operation robot 13 to theaccess position 601. In this case, by completing the movement of theoperation robot 13 before the pallet 15 reaches the access position 601,the operation robot 13 can execute the task B2 of unloading theoperation objects 16 from the pallet 15 immediately upon arrival of thepallet 15.

When the transport robot 11 arrives at the access position 601, in block804, the controller 610 can obtain one or more images of the operationobjects 16 and pallet 15 at the access position 601, such as using a3D-vision or other imaging sensor 522. The controller 610 can processthe resulting image data, and determine the states and situations of theoperation objects 16 and pallet 15. Furthermore, in block 804, based onthe information about the operation objects 16 and the information aboutthe pallet 15, the robot control unit 616 can create or derive a controlsequence, for example, by machine learning or the like. The processedinformation about the pallet 15 can include, for example, loadedsituation and the concentrated situation of the operation objects 16 onthe pallet. The control sequence can include designation of the positionof the operation objects 16 on the pallet 15 and/or a travel path of theend effector to the position for picking up and moving of the operationobject 16 from the access position 601 to the transport location 602.Based on the control sequence, the robot control unit 616 can instructthe operation robot 13 to execute the tasks B2 and B3 of gripping theoperation object 16 on the pallet 15. The pallet 15 can be held by thetransport robot 11 that may be stopped at the access position 601 and/ormoving to the transport location 602.

In that case, an index showing the handling property for unloading theoperation objects 16 from the pallet 15 may be calculated based on theinformation about the operation objects 16, the information about thepallet 15 (second data), and/or the processing result of the image data.Based on the index showing the handling property, the robot control unit616 may create or derive a control sequence by, for example, machinelearning or the like. The control sequence may specify the operationobjects 16 and the pallet 15 loaded with the operation objects 16. Inone embodiment, based on the index, the robot control unit 616 candetermine pallet information representing the one of the pallets 15 fromwhich the operation objects 16 are to be unloaded.

Furthermore, similar to the first task, the controller 610 can determinepositions of the transport robot 11 and pallet 15 from image data of thepallet 15 at the access position 601. The data deriving unit 612 canderive relative positional information about the transport robot 11 andthe pallet 15. The robot control unit 616 may correct positionalinformation about the pallet 15 at the access position 601 based on therelative positional information about the two Accordingly, the robotcontrol unit 616 can create or derive a control sequence for the taskB2, for example, by machine learning or the like, based on the correctedpositional information about the pallet 15.

In block 805, the controller 610 can obtain one or more images of theoperation objects 16 held at the transport location 602 as well asimages of the transport location 602 (e.g., a conveyor surface and finalposition), such as using a 3D-vision or other imaging sensor 522. Thecontroller 610 can process the resulting image data, and therebydetermine the states and situations of the operation objects 16 and thetransport location 602. Furthermore, in block 805, the robot controlunit 616 can create or derive a control sequence, for example, bymachine learning or the like. The control sequence can include aspecified position at which the operation objects 16 are transportedand/or travel paths of the operation objects 16 to the specifiedposition. Based on the control sequence, the robot control unit 616 caninstruct the operation robot 13 to execute the task B4 of dropping theoperation objects 16 at the specified position in the transport location602. Note that in directly placing the operation objects 16 in acontainer provided at the transport location 602, the availability ofspace in the container may be checked using the imaging sensor 522 toincrease the storage efficiency.

When the unloading of operation objects 16 from the pallet 15 iscomplete, in block 806, the robot control unit 616 can create or derivea control sequence, for example, by machine learning or the like. Thecontrol sequence can include a transport path for returning the pallet15 from which the operation objects 16 have been unloaded, from theaccess position 601 to the storage location in the storage area 101,such as using the transport robot 11. Based on the control sequence, therobot control unit 616 can instruct the transport robot 11 to executethe task B5 of transporting the pallet 15 to the storage area 101.

When the above-mentioned second task of unloading the operation objects16 from the pallet 15 has been completed, in block 807, the dataderiving unit 612 can derive the second data including information aboutthe pallet 15 from which the operation objects 16 have been unloaded.The data deriving unit 612 can also derive the second data includinginformation about the other operation objects 16 loaded on the pallet15, including information about the loaded locations of the otheroperation objects 16 on the pallet 15 and information about SKU thereof.The data storage unit 614 can store the second data in association withthe operation objects 16 and the pallet 15. In this way, the second datarepresentative of the loading situation of the operation objects 16 onthe pallet 15 can be updated for each operation object 16 and for eachunloading operation of the operation objects 16. Also, based on thesecond data, the data storage unit 614 can store the positionalinformation about multiple operation objects 16 on the pallet 15 astwo-dimensional information and/or three-dimensional information foreach operation object 16 and/or for each pallet 15.

Also, if the operation objects 16 have identification information on,for example, a side face, the identification information about theoperation objects 16 can be checked using the imaging sensor 522 evenwhile the operation objects 16 are stacked on the pallet 15. Thus, evenif the operation objects 16 remain stacked on the pallet 15, theoperation objects 16 can be identified easily by actual measurement.Furthermore, the imaging sensor 522 can be configured to image the shelf15 when the transport robot 11 holding the pallet 15 is at rest or whenthe transport robot 11 holding the pallet 15 is moving.

Also, the robot control unit 616 may be configured to determine orindicate that unloading of an operation object 16 from the pallet 15 iscompleted before the unloading work (task B2) of the operation object 16from the pallet 15 is actually finished. For example, by checking theoperating state of an end effector of the operation robot 13 or bykeeping track of movement of the operation object 16 using the imagingsensor 522, it can be determined that unloading of the operation object16 is completed. The robot control unit 616 can determine or indicatethe completion status when the entire operation object 16 or most partof the operation object 16 is moved out of the pallet 15 and/or when theentire operation object 16 or part of the operation object 16 leaves thepallet 15 or the operation object 16 loaded behind it.

The controller 610 and control method for the robotic system 500 or 600configured as described above make it possible to grasp informationabout the operation objects 16 and the pallet 15 before loading orunloading the operation objects 16 on or from the pallet 15. Thecontroller 610 can implement tasks (first and second tasks) that arebased on a control sequence created or derived, for example, by machinelearning or the like. The control sequence can be created or derivedbased on the first data including the above information. This in turnmakes it possible to efficiently and smoothly perform the operation ofloading the operation objects 16 on the pallet 15 or the operation ofunloading the operation objects 16 from the pallet 15. In so doing,sophisticated cooperation among units of the robotic system 500 or 600(e.g., between the transport robot 11 and operation robot 13) can beimplemented, making it possible to expand the functionality of therobotic systems 500 and 600.

Other Application Example

FIG. 9 is a schematic plan view showing an example environment in whicha robotic system according to one embodiment of the present technologycan operate. As with the robotic system 600, a robotic system 100 can beimplemented, for example, for a warehouse or the like, and implementoperations for warehousing, replenishment, storage, and/or shipping in alogistics system. Also, the robotic system 100 can be configured toperform or execute one or more tasks and/or an appropriate combinationof multiple tasks. The tasks can be performed or executed suitably bythe robotic system 100 via one or more units (e.g., various robots,various devices, a controller provided integrally therewith orseparately therefrom).

As shown in FIG. 9, the robotic system 100 can include one or moretransport robots 11 and 12 (e.g., a pallet AGV) serving as a transportunit/transport units. The robotic system 100 can further include one ormore operation robots 13 (e.g., piece-picking robots) serving astransporting/sorting units to move operation objects between differentplaces. The robotic system 100 can also include a controller (not shownin FIG. 9) for the operation robots 13 and/or the transport robots 11and 12.

To achieve predetermined purposes, the tasks executed by the roboticsystem 100 can be combined such that each unit will execute individualtasks in appropriate order or configured such that each unit willexecute various different tasks selected as appropriate. Each unit ofthe robotic system 100 can execute one or more tasks to access variousdifferent items located in the storage area 101, such as on or in thepallet 15 or a container 16. The robotic system 100 can access variousitems according to a predetermined incoming/packing order or accordingto properties of the items (e.g., SKU).

The transport robot 11 can execute the task of transporting, forexample, the pallets 15 in which containers 16 storing ordered items arestacked between the storage area 101 and a transportation area 102. Onthe other hand, the transport robot 12 can retrieve (pick up) containers16 storing ordered items from the transported pallets 15. Also, thetransport robot 12 can execute the task of transporting the containers16 between the transportation area 102 and the grouping area 103 (anarea used to group items: the stop location SL of the containers 16 inFIG. 9). The operation robot 13 can execute the task of picking upordered items from the containers 16 in the grouping area 103 and movingthe containers 16 to the transport location OL.

In executing a task, for example, when a shipment order is issued, whenstored objects are rearranged, or when item replenishment is carriedout, the robotic system 100 can identify different areas in whichindividual units and/or unit groups operate. For example, the roboticsystem 100 can identify the storage area 101 in which the transportrobot 11 operates, the transportation area 102 in which the transportrobot 12 operates, and/or the grouping area 103 in which the transportrobot 12 and/or operation robot 13 operate. Note that the areas in whichthe transport robots 11 and 12 and operation robot 13 operate may not belimited to the areas described above. For example, the transport robot11 may further operate in the transportation area 102 and the transportrobot 12 may further operate in the storage area 101 according to anappropriate control sequence.

Also, the areas in which the units operate may be adjacent to each otheror partially overlap each other. For example, the storage area 101 maybe adjacent to the transportation area 102 while the transportation area102 may partially overlap the grouping area 103. In that case, thetransport robots 11 and 12 can operate in different areas by accessingthe pallet 15 in the transportation area 102, for example, as shown inFIG. 9. Consequently, the robotic system 100 can reduce the possibilityof potential collisions between different types of unit and/or thepossibility of the different units obstructing each other. Note that theoperation robot 13 can be fixed in place, making it easy for thetransport robot 12 to enter the grouping area 103 without colliding withother units or causing congestion during movement.

Also, the robotic system 100 can use appropriate paths to navigatetransport units such as the transport robots 11 and 12. For example, therobotic system 100 can use a first path P1 to maneuver one or moretransport robots 11, and/or a second path P2 to maneuver one or moretransport robots 12. In some embodiments, for example, the first path P1and second path P2 may be separated from each other by a distance orspace so as not to overlap each other as shown in FIG. 9.

Also, floor markings (e.g., painting or tape) may be provided on thefirst path P1 and second path P2. The transport units (e.g., thetransport robots 11 and 12) can follow the floor markings when travelingby themselves. This allows the robotic system 100 to identifyshapes/positions of the floor markings and use the identifiedinformation when instructing the transport units. Accordingly, therobotic system and/or the transport units can perform path calculationsand/or do sequencing of positions (e.g., pick-up positions and/or droppositions of objects to be transported). Furthermore, the first path P1and second path P2 can include a series of links (e.g., paths) and nodes(e.g., crossing positions of paths or specified positions used inchanging a travel direction).

Also, the robotic system 100 can calculate efficient paths (e.g., pathsassociated with lowest cost metrics) that allow transport units to movebetween pick-up positions and/or drop positions of objects to betransported without interfering with other units. These transport unitscan calculate current positions on the paths using, for example, aposition/navigation system and move along specified paths and routesbased on the current positions.

Note that as described above, the robotic system 100 can set tasks as aseries of tasks executed by multiple units. The robotic system 100 canadjust and control each of the units based on characteristics ofindividual tasks such that execution of an integrated task can beoptimized. For example, the robotic system 100 can make adjustments suchas by updating a range of operations involving units or by setting arange of operations that involves plural specified areas.

As an illustrative example, to respond to a shipment order and fulfillthe order, tasks or an operation made up of an appropriate combinationof integrated tasks can be executed. The tasks may include: (1)pinpointing the storage location of ordered items in the storage area101, such as a pallet 15 loading ordered items or a pallet 15 on which acontainer 16 storing the ordered items is loaded; (2) transporting thepallet 15 to the transportation area 102; (3) transporting the container16 from the pallet 15 to the grouping area 103; (4) transporting theordered items from the container 16 to a target position such as ashipment container and/or conveyor; (5) returning the container 16 tothe pallet 15; and/or (6) returning the pallet 15 to the storage area101.

Also, to deal with warehousing of items or replenishment with items,tasks or an operation made up of an appropriate combination ofintegrated tasks can be executed. The tasks may include: (1) determininga container 16 capable of storing the items and a pallet 15 capable ofloading the container 16 as a storage location of the items based, forexample, on demand forecasting to determine an available storagelocation; (2) storing warehoused items or replenishment items in thecontainer 16; (3) loading the container 16 storing the warehoused itemsor replenishment items into the pallet 15; and/or (4) transporting thepallet 15 to an appropriate predetermined storage location.

Also, to maneuver various units, the robotic system 100 can generate,for example, commands, control settings, motion plans, positions,identification information, paths, current positions of the units,states, progress information, or a combination thereof needed to operateactuators in the respective units. The generated information can becommunicated among the individual units, and tasks can be executed bysharing the generated information among the controller and units.

Note that whereas in the above description of the robotic system 100,execution of tasks related to shipping (shipment) of items has mainlybeen described. In some embodiments, for example, a control sequence foroperating the robotic system 100 may be applied or adjusted (e.g., byreversing the above procedures and/or via creating or deriving newresults) to execute tasks related to warehousing orreplenishing/restocking items.

Various Example Embodiments

Embodiment 1 may relate to a controller for a robotic system. Thecontroller comprising: at least one processor; and at least one memorydevice connected to the at least one processor. The at least one memorydevice including instructions thereon that, when executed by the atleast one processor, cause the at least one processor to implement amethod for:

-   -   deriving first data and/or second data;        -   the first data representing one or more operation objects            and one or more pallets before loading or unloading the one            or more operation objects with respect to one of the one or            more pallets, and        -   the second data including representing the one or more            operation objects and the one or more pallets after the            loading or unloading the one or more operation objects with            respect to the one of the one or more pallets;    -   deriving a control sequence for loading and/or unloading an        operation object on or from one of the one or more pallets,        wherein the control sequence is derived based on the first data        and/or the second data, before implementing the task of the        loading and/or unloading; wherein the control sequence is        configured such that operation objects that correspond to        different stock keeping units are loaded on the one of the one        or more pallets; and    -   instructing an operation robot to execute a task of loading        and/or unloading the operation object on and/or from the one of        the one or more pallets based on the control sequence.

Embodiment 2 may include the controller of embodiment 1. In thisembodiment, deriving the control sequence may include deriving thecontrol sequence based on the first data and/or the second data suchthat operation objects including same or similar stock keeping units areplaced on the one of the one or more pallets in a concentrated manner.

Embodiment 3 may include the controller of embodiment 1 or 2. In thisembodiment, deriving the control sequence may include determining palletinformation based on the first data and/or the second data, the palletinformation representing the one of the one or more pallets which theoperation object is to be loaded on and/or unloaded from.

Embodiment 4 may include the controller of any one of embodiments 1-3.In this embodiment, the at least one memory device may further includeinstructions for calculating, based on the second data, an indexrepresenting handling property for unloading the operation object fromthe one of the one or more pallets, and wherein deriving the controlsequence may include determining pallet information based on the index,the pallet information representing the one of the one or more palletsfrom which the operation object is to be unloaded.

Embodiment 5 may include the controller of any one of embodiments 1-4.In this embodiment, the second data may represent positional informationof the one or more operation objects loaded on the one or more pallets.The positional information comprises two-dimensional information orthree-dimensional information for each operation object and/or for eachpallet.

Embodiment 6 may include the controller of any one of embodiments 1-5.In this embodiment, deriving the control sequence may include derivingthe control sequence for loading the operation object on the one of theone or more pallets so as to allow a sensor to measure identificationinformation of the operation object.

Embodiment 7 may include the controller of any one of embodiments 1-6.In this embodiment, the at least one memory device may further includeinstructions that cause the at least one processor for determining thatthe loading of the operation object on the one of the one or morepallets and/or the unloading of the operation object from the one of theone or more pallets are/is completed, before the actual execution ofloading and/or the unloading of the operation object are/is finished.

Embodiment 8 may relate to a tangible, non-transitory computer-readablemedium including processor instructions stored thereon that, whenexecuted by at least one processor thereof, cause the robotic system,the processor instructions comprising:

-   -   deriving first data and/or second data;        -   the first data representing information in connection with            one or more operation objects and information in connection            with one or more pallets before loading of the one or more            operation objects on one of the one or more pallets and/or            unloading of the one or more operation objects from one of            the one or more pallets, and        -   the second data representing information in connection with            the one or more operation objects and information in            connection with the one or more pallets after the loading of            the one or more operation objects on the one of the one or            more pallets and/or the unloading of the one or more            operation objects from the one of the one or more pallets;    -   deriving a control sequence for loading an operation object on        one of the one or more pallets and/or unloading the operation        object from one of the one or more pallets based on the first        data and/or the second data, before the loading of the operation        object on the one of the one or more pallets and/or the        unloading of the operation object from the one of the one or        more pallets; wherein the control sequence is configured such        that operation objects that correspond to different stock        keeping units are loaded on the one of the one or more pallets;        and    -   instructing an operation robot for executing a task of loading        the operation object on the one of the one or more pallets        and/or unloading the operation object from the one of the one or        more pallets based on the control sequence.

Embodiment 9 may include the tangible, non-transitory computer-readablemedium of embodiment 8. In this embodiment, deriving the controlsequence may include deriving the control sequence based on the firstdata and the second data for placing the operation objects having sameor similar stock keeping units on the one of the one or more pallets ina concentrated manner.

Embodiment 10 may include the tangible, non-transitory computer-readablemedium of embodiment 8 or 9. In this embodiment, deriving the controlsequence may include determining pallet information, based on the firstdata and the second data, the pallet information representing the one ofthe one or more pallets which the operation object is to be loaded onand/or unloaded from.

Embodiment 11 may include the tangible, non-transitory computer-readablemedium of any one of embodiments 8-10. In this embodiment, the processorinstructions may further comprise calculating, based on the second data,an index representing handling property for unloading the operationobject from the one of the one or more pallets, wherein deriving thecontrol sequence may include determining pallet information based on theindex, the pallet information representing the one of the one or morepallets from which the operation object is to be unloaded.

Embodiment 12 may include the tangible, non-transitory computer-readablemedium of any one of embodiments 8-11. In this embodiment, the seconddata may represent positional information of the one or more operationobjects loaded on the one or more pallets, the positional informationcomprising two-dimensional information or three-dimensional informationfor each operation object or for each pallet.

Embodiment 13 may include the tangible, non-transitory computer-readablemedium of any one of embodiments 8-12. In this embodiment, deriving thecontrol sequence may include deriving the control sequence for loadingthe operation object on the one of the one or more pallets so as toallow a sensor to measure identification information of the operationobject.

Embodiment 14 may relate to a method for controlling a robotic system.The method may comprise:

-   -   deriving first data and/or second data;        -   the first data representing one or more operation objects            and/or one or more pallets before loading of the one or more            operation objects on one of the one or more pallets and/or            unloading of the one or more operation objects from one of            the one or more pallets, and        -   the second data representing the one or more operation            objects and/or the one or more pallets after the loading of            the one or more operation objects on the one of the one or            more pallets and/or the unloading of the one or more            operation objects from the one of the one or more pallets;    -   deriving a control sequence for loading an operation object on        one of the one or more pallets and/or unloading the operation        object from one of the one or more pallets, wherein the control        sequence is derived based on the first data and the second data,        and derived before the loading of the operation object on the        one of the one or more pallets and/or the unloading of the        operation object from the one of the one or more pallets;        wherein the control sequence is configured such that operation        objects that correspond to different stock keeping units are        loaded on the one of the one or more pallets; and    -   instructing, based on the control sequence, an operation robot        to execute a task of loading the operation object on the one of        the one or more pallets and/or unloading the operation object        from the one of the one or more pallets.

Embodiment 15 may include the method of embodiment 14. In thisembodiment, deriving the control sequence may include deriving thecontrol sequence based on the first data and the second data for placingthe operation objects having same or similar stock keeping units placedon the one of the one or more pallets in a concentrated manner.

Embodiment 16 may include the method of embodiment 14 or 15. In thisembodiment, deriving the control sequence may include determining palletinformation based on the first data and the second data, the palletinformation representing the one of the one or more pallets which theoperation object is to be loaded on and/or unloaded from.

Embodiment 17 may include the method of any one of embodiments 14-16. Inthis embodiment, the method may further comprise calculating, based onthe second data, an index representing handling property for unloadingthe operation object from the one of the one or more pallets, whereinderiving the control sequence may include determining pallet informationbased on the index, the pallet information representing the one of theone or more pallets from which the operation object is to be unloaded.

Embodiment 18 may include the method of any one of embodiments 14-17. Inthis embodiment, the second data may represent positional information ofthe one or more operation objects loaded on the one or more pallets. Thepositional information comprises two-dimensional information orthree-dimensional information for each operation object or for eachpallet.

Embodiment 19 may include the method of any one of embodiments 14-18. Inthis embodiment, deriving the control sequence may include deriving thecontrol sequence for loading the operation object on the one of the oneor more pallets so as to allow a sensor to measure identificationinformation of the operation object.

Embodiment 20 may include the method of any one of embodiments 14-19. Inthis embodiment, the method may further comprise determining a statusrepresenting loading of the operation object on the one of the one ormore pallets and/or the status representing unloading of the operationobject from the one of the one or more pallets are/is completed, beforethe loading and/or the unloading of the operation object are/isfinished.

CONCLUSION

Whereas embodiments as examples of the present technology have beendescribed in detail above, the above descriptions are merelyillustrative of the present technology in all respects and are intendedto facilitate the understanding of the present invention, but are not tobe interpreted as limiting the present invention. Besides, needless tosay, various improvements and modifications can be made withoutdeparting from the scope of the present technology, and the componentsof the embodiments as well as the arrangements, materials, conditions,shapes, sizes, and the like of the components are not limited to thoseillustrated above or specific ones, and may be changed as appropriate.

In other words, embodiments of the present technology do not excludeother configurations, or limit the present technology to the aboveembodiments. Modified forms equivalent to the embodiments of the presenttechnology can be implemented within the scope of the presenttechnology. For example, processes, steps, routines, or blocks can becarried out in different orders in alternative embodiments within thescope of the present technology, and some processes, steps, routines, orblocks, may be deleted, moved, added, subdivided, combined, and/ortransformed within the scope of the present technology. Also, theprocesses, steps, routines, or blocks may be carried out by variousdifferent methods. Furthermore, even if the processes, steps, routines,or blocks are to be carried out successively in the above embodiments,the processes, steps, routines, or blocks can be carried outconcurrently in some cases or may be carried out non-successively atdifferent times. Furthermore, specific numerical figures cited hereinmay be different values or different ranges.

Also, the above embodiments may be carried out by many alternativemethods.

Furthermore, details of the above embodiments may be changedsignificantly in specific modes of implementation, but are stillincluded in the techniques of the present technology. In addition, thespecific terms used to describe specific features or aspects of thepresent technology are not limited to specific properties, features, oraspects, or specific embodiments which are used in the presenttechnology and with which the terms are associated, and thus the presentinvention is not limited to specific meanings except as defined in theappended claims. Also, whereas the present invention is defined by anarbitrary number of claims, needless to say, various aspects areexpected within the scope of the present technology.

I/We claim:
 1. A controller for controlling an operation robot tomanipulate an operation object to load the operation object on a palletand/or unload the operation object from the pallet, the controllercomprising: a data deriving unit for deriving: first data includinginformation about the operation object and information about the palletbefore loading of the operation object on the pallet and/or beforeunloading of the operation object from the pallet, and second dataincluding information about the operation object and information aboutthe pallet after loading of the operation object on the pallet and/orafter unloading of the operation object from the pallet; a data storageunit for storing the first data and the second data; and a robot controlunit for: creating or deriving a control sequence for loading of theoperation object on the pallet and/or unloading of the operation objectfrom the pallet based on the first data before loading of the operationobject on the pallet and/or before unloading of the operation objectfrom the pallet, and instructing the operation robot to, based on thecontrol sequence, execute a task of loading the operation object on thepallet and/or unloading the operation object from the pallet, whereinthe robot control unit creates or derives the control sequence based onthe first data and the second data such that: a plurality of types ofoperation objects in which stock keeping units are different from eachother are loaded on the pallet, and operation objects having same orsimilar stock keeping units are placed on the pallet in a concentratedmanner.
 2. The controller of claim 1, wherein the robot control unit:calculates, based on the second data, an index indicating handlingproperty for unloading of the operation object from the pallet, andbased on the index, creates or derives the control sequence includingidentification of the operation object and the pallet loaded with theoperation object.
 3. The controller of claim 1, wherein the data storageunit stores positional information about a plurality of the operationobjects on the pallet as two-dimensional information orthree-dimensional information for each operation object of the pluralityor for each pallet.
 4. The controller of claim 1, wherein: the robotcontrol unit creates or derives the control sequence for loading theoperation object on the pallet so as to allow a sensor to measureidentification information about the operation object, and the sensor isconfigured for imaging of the operation object or of the pallet loadedwith the operation object.
 5. The controller of claim 1, wherein, beforeloading of the operation object on the pallet and/or unloading of theoperation object from the pallet are/is finished, the robot control unitdetermines that the loading of the operation object on the pallet and/orthe unloading of the operation object from the pallet are/is completed.6. The controller of claim 1, wherein the data deriving unit derivesmeasured values or estimated values of the information about theoperation object and/or of the information about the pallet.
 7. Thecontroller of claim 1, wherein: the controller identifies (i) a roboticsystem that includes the operation robot and (ii) a region related tothe robotic system, and calculates tasks based on the control sequence;and the tasks include a task related to transportation of the pallet andoperating of the operation object by the robotic system and a pluralityof tasks to be executed by the robotic system across adjacent and/oroverlapping regions.
 8. A tangible, non-transitory computer readablemedium having processor instructions stored thereon that, when executedby one or more processors of a computer, cause the computer to perform amethod for controlling an operation robot for manipulating an operationobject to load the operation object on a pallet and/or unload theoperation object from the pallet, and wherein the method comprises:deriving first data including information about the operation object andinformation about the pallet before loading of the operation object onthe pallet and/or before unloading of the operation object from thepallet; deriving second data including information about the operationobject and information about the pallet after loading of the operationobject on the pallet and/or after unloading of the operation object fromthe pallet; storing the first data and the second data; creating orderiving a control sequence for loading of the operation object on thepallet and/or unloading of the operation object from the pallet based onthe first data before loading of the operation object on the palletand/or before unloading of the operation object from the pallet; andinstructing the operation robot to, based on the control sequence,execute a task of loading the operation object on the pallet and/orunloading the operation object from the pallet, wherein creating orderiving the control sequence includes creating or deriving the controlsequence based on the first data and the second data such that: aplurality of types of operation objects in which stock keeping units aredifferent from each other are loaded on the pallet, and operationobjects having same or similar stock keeping units are placed on thepallet in a concentrated manner.
 9. The tangible, non-transitorycomputer-readable medium of claim 8, wherein: the method furthercomprises calculating, based on the second data, an index indicatinghandling property for unloading of the operation object from the pallet,and creating or deriving the control sequence includes creating orderiving, based on the index, the control sequence includingidentification of the operation object and the pallet loaded with theoperation object.
 10. The tangible, non-transitory computer-readablemedium of claim 8, wherein the method further comprises storingpositional information about a plurality of the operation objects on thepallet as two-dimensional information or three-dimensional informationfor each operation object of the plurality or for each pallet.
 11. Thetangible, non-transitory computer-readable medium of claim 8, wherein:creating or deriving the control sequence includes creating or derivingthe control sequence for loading the operation object on the pallet soas to allow a sensor to measure identification information about theoperation object, and the sensor is configured for imaging of theoperation object or of the pallet loaded with the operation object. 12.The tangible, non-transitory computer-readable medium of claim 8,wherein the method further comprises, before loading of the operationobject on the pallet and/or unloading of the operation object from thepallet are/is finished, determining that the loading of the operationobject on the pallet and/or the unloading of the operation object fromthe pallet are/is completed.
 13. The tangible, non-transitorycomputer-readable medium of claim 8, wherein the method furthercomprises deriving measured values or estimated values of theinformation about the operation object and/or of the information aboutthe pallet.
 14. A method for controlling an operation robot adapted tomanipulate an operation object to load the operation object on a palletand/or unload the operation object from the pallet, wherein the methodis performed by a controller having a data deriving unit, a data storageunit, and a robot control unit, and wherein the method comprises:deriving first data using the data deriving unit, the first dataincluding information about the operation object and information aboutthe pallet, before loading of the operation object on the pallet and/orbefore unloading of the operation object from the pallet; storing thefirst data using the data storage unit; creating or deriving, using therobot control unit and based on the first data, a control sequence forloading an operation object on the pallet and/or unloading an operationobject from the pallet before loading of the operation object on thepallet and/or unloading of the operation object from the pallet; andinstructing, using the robot control unit and based on the controlsequence, the operation robot to execute a task of loading the operationobject on the pallet and/or unloading the operation object from thepallet.
 15. The method of claim 14, further comprising: deriving seconddata using the data deriving unit, the second data including informationabout the operation object and information about the pallet, afterloading of the operation object on the pallet and/or after unloading ofthe operation object from the pallet; and calculating, using the robotcontrol unit and based on the second data, an index indicating handlingproperty for unloading of the operation object from the pallet, whereincreating or deriving the control sequence includes creating or deriving,based on the index, the control sequence including identification of theoperation object and the pallet loaded with the operation object. 16.The method of claim 14, further comprising storing, using the datastorage unit, positional information about a plurality of the operationobjects on the pallet as two-dimensional information orthree-dimensional information for each operation object of the pluralityof the operation objects or for each pallet.
 17. The method of claim 14,wherein the creating or deriving the control sequence includes creatingor deriving the control sequence for loading the operation object on thepallet so as to allow a sensor to measure identification informationabout the operation object.
 18. The method of claim 14, furthercomprising determining that loading of the operation object on thepallet and/or unloading of the operation object from the pallet are/iscompleted, before the loading of the operation object on the palletand/or the unloading of the operation object from the pallet are/isfinished.
 19. The method of claim 14, further comprising deriving, usingthe data deriving unit, measured values or estimated values of theinformation about the operation object and/or of the information aboutthe pallet.
 20. The method of claim 14, further comprising: identifyinga robotic system that includes the operation robot and a region relatedto the robotic system; and calculating tasks based on the controlsequence, wherein the tasks include (i) a task related to transportationof the pallet and manipulation of the operation object by the roboticsystem and (ii) a plurality of tasks to be executed by the roboticsystem across adjacent and/or overlapping regions.